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• Background 

• Simulation Credibility Level (SCL) Scale 



- Reports on the credibility of a product of M&S 
technology, namely, a simulation result. 

- Represent milestones demarking progress towards 
achieving credible simulations. 

• Credibility Assessment Scale 

- Assesses the rigor of the processes used to produce 
the simulation results and determine their favorability 
against key factors that affect the credibility judgment. 

- Requires extensive documentation. 

A • Summary 
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How accurate are simulations and test data? 

What is the level of confidence in the design 
based on these simulations and data? 

Are system requirements met? 


Decision-Maker 
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Simulation and Test Debate 





Test data are the reality. 

Simulation models are questionable. 
Simulations are generated with many knobs. 
Simulation uncertainties are not known. 


Tester 

(One who conducts tests) 


r 




Simulations represent physics. 

The data reduction procedure is questionable 
The flow environment in the test is unknown. 
Test uncertainties are questionable. 



Simulator 

(One who 
simulates) 




Both wind tunnels and CFD are “notorious liars.” 

— Paul Rubbert 
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Sources of Uncertainties in 
Simulations and Tests 
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Equivalence 

Accuracy 

Isolation of Phenomena 
Extraneous Phenomena 
Modeling 

Creativity over belief 


Insufficiency of data 
Accuracy 

Isolation of Phenomena 
Extraneous Phenomena 
Creativity over belief 
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Simulation Credibility 


Simulation 

Uncertainties 
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( Lack of equivalence 
Lack of accuracy 


Simulation 

Precision 


I Verification 


1 Isolation of phenomena i Human Error 
Extraneous phenomena | Creativity 
Modeling of phenomena t Over belief 


I Validation 


Simulation 

Bias 


Personal 

Bias 


I Independent 
\ Review 


Numerics 

Uncertainty 


Physics 

Uncertainty 


Certification 











Simulation 

Certification 


Validation 

for 

Prediction 


Validation 

for 

Postdiction 


Simulation 

Verification 


Model 

Verification 
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Simulation Credibility Scale 
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weaknesses are certified for making 
simulation-based decisions. 

Reality models are validated, their uncertainties 
are quantified, and both are certified for 
predictions. 

Reality models are validated, their uncertainties 
are quantified, and both are certified for 
postdictions. 

Numerical uncertainties and sensitivities are 
quantified and certified. 


Conceptual, mathematical, and simulation 
models are verified and certified. 


0 

SCL 


7 


Postdiction and Prediction 


Physics Complexity 

Prediction 

Domain 

Inference 

Postdiction 

Domain 

A 
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SCL 1 : Model Verification - 
Supporting Information 

• Verified 

- Simulation requirements 

- The conceptual model 

- The transformation of the conceptual model to the 
mathematical model 

- The equivalence of the simulation model to the 
mathematical model 

- The correctness of various coded phenomena 
individually and of complex couplings of various 
phenomena. 

- The correctness of simulation code 
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SCL 2: Simulation Verification - 
Supporting Information 

• Numerical uncertainties 

- All relevant sources are identified. 

- A plan to manage and reduce uncertainties is developed. 

- Uncertainties and their sensitivities are quantified. 

- Levels of confidence in quantified uncertainties are 
estimated. 




Whether achieved uncertainties meet the 
requirements is determined. 

Recipes for generating simulations are developed. 
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SCL 3: Validation for Postdiction - 
Supporting Information 

• Uncertainties 

- The same supporting information as that for SCL2. 

• Validation with real-world data 

- The technique of Design of Experiment is used to 
conduct reality tests for validations. 

- Simulations are generated before tests are 
conducted. 

- No reality data are used for validation without knowing 
their uncertainties. 

- Numerical uncertainties are appreciably smaller than 
reality data uncertainties. 

• Recipes for generating postdictions are 
developed. 
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SCL 4: Validation for Prediction - 
Supporting Information 

• Th^ppropriateness of models and numerics is 
reassessed and justified, and limitations identified. 

• Methods are developed and justified to quantify 
uncertainties in the prediction domain. 

• Uncertainties 

- The same supporting information as that for SCL2. 

• The technique of Design of Experiment is used to conduct 
simulations for validation with circumstantial evidence. 

• Whether uncertainties meet the requirements is 
determined. 

• The conceptual model is validated. 

• Recipes for generating predictions are documented. 
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SCL 5: Simulation Certification - 
Supporting Information 



The acceptability of simulations is justified. 

The certification authority is also provided 

- Simulations along with their relevant uncertainties 
and confidence levels 

- Previous certifications and previous outcomes of 
reviews at lower SCLs are also provided. 
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Credibility Assessment Scale 







Key Aspects of Credibility Levels 




Level 

Veiification 

Validation 

Input Pedisi ee 

Results 

Uncertain^' 

Results 

Robustness 

Use Histoiy 

M&S 

Management 

People 

Qualifications 

4 

Numerical 
errors small 
for all 
in^rtant 
features. 

Results agree 
with real- 
world data. 

Input data agree 
with real-world 
data. 

Non- 

deterministic 
& numerical 
analysis. 

Sensitivity 
known for 
most 

parameters; 

key 

sensitivities 

identified. 

De facto 
standard. 

Continual 

process 

in^rovement. 

Extensive 
experience in 
and use of 
recommended 
practices for this 
particular M&S. 

3 

Formal 

numencal 

error 

estimadoii. 

Results agree 
with 

experimental 
data for 
problems of 
interest. 

Input data agree 
with 

experimental 
data for 
problems of 
mterest. 

Non- 

deterministic 

analysis. 

Sensitivity 
known for 
many 

parameters. 

Previous 
predictions 
were later 
validated by 
mission data. 

Predictable 

process. 

Advanced 
degree or 
extensive M&S 
experience, and 
recommended 
practice 
knowledge. 

2 

Unit and 
regression 
testing of 
key features. 

Results agree 
with 

experimental 
data or other 
M&S on unit 
problems. 

Input data 
traceable to 
formal 

documentation. 

Deterministic 
analysis or 
expert 
opinion. 

Sensitivity 
known for a 
few 

parameters. 

Used before 
for critical 
decisions. 

Estabhshed 

process. 

Formal M&S 
training and 
experience, and 
recommended 
practice trainmg. 

1 

Conceptual 

and 

mathematical 

models 

verified. 

Conceptual 

and 

mathematical 
models agree 
with smqjle 
referents. 

Input data 
traceable to 
informal 
documentation. 

Quahtative 

estimates. 

Qualitative 

estimates. 

Passes 
sinq)le tests. 

Managed 

process. 

Engineering or 
science degree. 

0 

Insufficient 

e\idence. 

Insufficient 

evidence. 

Insufficient 

evidence. 

Insufficient 

evidence. 

Insufficient 

evidence. 

Insufficient 

e\^dence. 

Insufficient 

evidence. 

Insufficient 

evidence. 


M&S Development 

M&S Operations 

Supporting Evidence 
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Level Definitions 



Level 

Verification E\1dence 

Validafiou E>1dence 

4 

Reliable error estimation methods are used to 
quantitatively assess numerical errors. Tliese 
estimates show that the enors are small from test 
suites, which exercise all important algorithms, all 
unportant featmes and capabilities, and all 
uuportant couplings (physics, modules, etc.) of the 
frill computational model. 

M&S results compar e favorably for the real- 
world system at validation points by comparison 
of M&S results to an acceptable referent, which 
is measurements on the real-world system. 

3 

Some foinial method is used to assess niunerical 
enors associated with unit testing with significant 
coverage of the code. 

M&S results compar e favorably for problems of 
interest at validation pomts by comparison of 
M&S results to an acceptable referent, which is 
experimental measurements on problems of 
interest. 

2 

Favorable results from unit and regression testing 
of key features of the computational model. 

M&S results compare favorably for miit 
problems at validation pomts by comparison of 
M&S results to an acceptable referent, which is 
either experimental measurements or higlier- 
fidelity M&S results. 

1 

Favorable evidence of verification for conceptual 
and mathematical models. 

M&S conceptual and mathematical models 
compar e favorably with “general problem” and 
“textbook” referents. 

0 

hisufficient evidence. 

Insufficient evidence. 
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Roll-Up Of Sub-Factor Score 


Siibfactor 

Siibfactor 

Weight 

Assessed Score 

Weighted 

Score 

Factor Score 

Validation 

Evidence 

0.7 

3 

2.1 

3.3 

Validation 

Teclmical 

Review 

0.3 

4 

1.2 
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Roll-Up of Factor Scores to Overall 

Score 



Facto 1 ’ 

Factor Score 

Overall 

Score 

Verification 

3 


Validation 

3.3 


Input Pedigree 

3.3 


Results Uncertainty 

3 

1.7 

Results Robustness 

1.7 

Use Histoiy 

4 


M&S Management 

3 


People Qualifications 

3 
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Summary 
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Simulation Credibility Level (SCL) Scale 

- The focus is on simulation result. 

- Assesses simulation credibility vis-a-vis the quantitative 
requirement on the result. 

- A la TRL, SCL demarks progress towards achieving the 
most credible simulation. 

Credibility Assessment Scale 

- The documentation burden and the credibility 
assessment are the two main shortcomings of this 
approach. 

• The focus is on processes. 

• Uses Analytical Hierarchy Process for roll-up of credibility score. 

• Decouples validation from verification. 
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